ECE 5725: Raspberry Pi Drum Tutor

Xiaotian Fang (xf233) and Demian Yutin (dy245)
Fall 2023

Demonstration Video

Introduction

In this project, we aimed to harness the power of embedded systems to create an interactive and portable electronic drum kit. Central to our design was the integration of pressure sensors within each drum pad, capable of capturing the dynamic range of a drummer's strike and translating it into a symphony of digital beats. Upon impact, these sensors generate analog signals that are then converted into digital form via an ADC, allowing a Raspberry Pi to produce a corresponding drum sound. To enrich the tactile feedback, we incorporated LEDs that flash in sync with the drum hits, augmenting the sensory experience of the drummer.

The project is further augmented with an educational tutorial system and an engaging game mode. The tutorial system is stratified into three levels of difficulty—easy, medium, and hard—to cater to a wide spectrum of learners. Each drum pad is connected to a PiTFT display that shows musical notation, with a cursor that glides across the score, cueing the user on when to strike. Accompanying the visual guide, the system also plays back the selected tracks, inviting users to match their rhythm with the music.

The game mode transforms drumming into an immersive rhythm game, reminiscent of popular music games. Five keys, representing different drum sounds, challenge users to hit the correct drum pad as notes scroll down to the target area, much like playing a melody in a dance of precision and timing. Successful hits score points, while misses lead to deductions. The movement and timing of the notes are meticulously mapped to the rhythm and tempo of each song, pushing users to not only play the drums but to feel and master the pulse and flow of the music.

This blend of hardware innovation and software creativity elevates the traditional electronic drum kit into a multifaceted platform for both entertainment and education, emphasizing rhythm accuracy, hand-eye coordination, and the sheer joy of drumming.

Project Objective:

Sensory Feedback and Interaction: To make sure our drum kit can accurately capture the force of drum strikes, converting them into authentic drum sounds, accompanied by visual feedback through synchronized LED lights.
Tutorial System: The system will visually display drum scores on piTFT and provide auditory cues through rhythm tracks, guiding users to play along and improve their timing and musicality.
Gaming Experience: The game will feature moving notes corresponding to different drums, requiring precise timing to achieve high scores, thereby fostering a deeper sense of rhythm and coordination.

Design and Testing

Our initial design process was to think of all the physical components in our system and how they might connect. This was initially described in this diagram:

Block diagram of components and connections: touch screen, RPi, drum pad, LED strip, speaker, power strip, drum sticks

Initial design diagram

After we listed all the components of the system, we began working on one part at a time, starting with the drum pad. Prof. Skovira generously provided us with a non-working Guitar Hero XBox drum pad, which we decided to base our system on.

The first week was spent on disassembling the drum pad, removing unnecessary components, replacing missing ones, replacing and resoldering the wires, and testing that the piezoelectric sensors were able to pick up drum hits using an oscilloscope.

Inside of drum pad chassis with colored wires going from sensors to a hole in the middle

Sensor lead wires replaced and rerouted

Drum pad with RPi in center, where the XBox controller was

Reassembled system with RPi in position

Oscilloscope reading of four drum hits, showing consistent voltage spike

Oscilloscope reading of voltage across a sensor for four drum hits

After this, we connected an analog-digital converter (the MCP3008) to our circuit to read the sensor voltages with the Raspberry Pi, using this forum thread and this documentation for reference. We used this source from the Adafruit website to guide us in writing a Python script to read the voltages. Because our system included the PiTFT touchscreen, which already utilizes the default SPI pins, we had to change this code to use a different SPI channel for the MCP3008, and had to add spi1 to the device tree in /boot/config.txt.

After this we worked on the software side of the project. This included the UI, the tutorial mode, the game mode, and the audio/visual feedback system. To make the design of this system manageable, we worked on each part separately, and combined the separate components after each was working individually.

The touch-screen UI and game mode were written using PyGame. For the tutorial mode, we played videos using mplayer, sending commands to a fifo to start/stop the video as necessary. To detect drum hits, we polled the voltage reading sent by the ADC every 1ms in a separate thread, and when a voltage spike was detected, we posted a PyGame event which was handled in the main thread. To give the player feedback on their drum hits, whenever a drum event was detected, we would play a sound using PyGame and light up the corresponding LED by setting a GPIO pin high.

The LEDs were added in the final week since we didn't have the parts before then. Since the power-on voltage of each LED was 12 volts, we wired each of them through a MOSFET so we could use the RPi's 3.3 volt GPIO output to switch the current supplied by a 12V power supply. To soften the bright light produced by the LEDs, we wrapped each LED with a paper cylinder to act as a diffuser.

Final version of hardware wiring

Results

We are pleased to report the successful completion of all the primary objectives set out for our electronic drum project. Our system has reached a level of performance that meets our initial goals, providing a responsive and interactive platform for drummers to practice, learn, and play.

The integration of pressure sensors into each drum pad has resulted in a highly responsive system that accurately captures the force of the drummer's strike, converting it into digital signals without noticeable delay. This allows for an authentic replication of acoustic drumming in a digital format, which is further enhanced by the clear and high-fidelity drum sounds produced by the system.

Our tutorial system has been well-received, offering three levels of difficulty to accommodate drummers at different skill levels. The visual and auditory cues provided by the system effectively guide users through each drumming session, improving their timing and technique.

The game mode, inspired by rhythm games, has been a standout feature, challenging users to keep the beat with moving notes that must be matched with timely drum strikes. This mode not only tests the player's rhythm and timing but also provides an engaging and fun way to practice drumming, which has proven to be a favorite among users.

Conclusion

In conclusion, our electronic drum project has achieved its intended targets, resulting in a versatile and user-friendly system that bridges the gap between traditional drumming and digital music production. We look forward to seeing how drummers of all levels will use this system to enhance their musical prowess and enjoy the art of drumming in a modern, technologically advanced format.

Future Work

For future work, we wanted to significantly enhance the functionality and educational value of our electronic drum system. Here are the proposed developments:

Decoding MIDI Files for Game Mode: We aim to implement a feature that can decode MIDI files to generate the drum notes for the game mode. This will ensure that the notes users play along with are not random but are accurate representations of actual drum scores. This capability will provide a more authentic drumming experience and can serve as a valuable learning tool for users to practice real drum pieces.

Interactive LED Guidance Module: We propose to develop a guidance module that utilizes the LED system not just for feedback but for proactive guidance. In tutorial mode, the LEDs will light up in synchronization with the cursor on the drum notation display, indicating which drum pad the user should strike at each beat. This feature aims to assist users in learning drum patterns more effectively by providing a visual guide that anticipates the next drum stroke.

Work Distribution

Project group picture

Demian Yutin

dy245@cornell.edu

Hardware assembly and soldering

Improved audio playback and drum hit detection

Code architecture design

Code module integration

Testing

Xiaotian Fang

xf233@cornell.edu

Game mode implementation

Hardware assembly and soldering

Improving video playback and return logic

UI design

Testing

Budget

Raspberry Pi $35.00

Drum Sticks - $14.99

LED Strips - $9.99

Drumset, MOSFETs, Resistors and Wires - Provided in lab

Total: $59.98

References

Pi Augmented Reality
Pigpio Library
R-Pi GPIO Document
Sensor circuit with MCP3008
MCP3008 datasheet
Raspberry Pi Cookbook, First Edition, pg. 210 (MOSFET wiring)

Code Appendix

# adc
import os
import time
import busio
import digitalio
import board
import adafruit_mcp3xxx.mcp3008 as MCP
from adafruit_mcp3xxx.analog_in import AnalogIn
# sound, UI
import pygame
import sys
# other
import RPi.GPIO as GPIO
import random
from threading import Thread
from time import sleep
import subprocess
# custom
from button import *

# ============================= GLOBAL CONSTANTS =================
# switch for whether we're running on the RPi or on a computer
ON_THE_PI = True
# pins used on the board
LED_PINS = [2,3,4,6,13]
QUIT_BUTTON = 27
RETURN_BUTTON=23
SHUTDOWN_BUTTON=22
# colors
WHITE = (255, 255, 255)
BLACK = (0, 0, 0)
RED = (255, 0, 0)
GREEN = (0, 255, 0)
BLUE = (0, 0, 255)
YELLOW = (255, 255, 0)
COLORS = [RED, YELLOW, BLUE, WHITE, GREEN]
# screen dimensions
screen_width, screen_height = 320, 240
# buttons of different menus
buttons = [
  [Button("Start", 160, 100), Button("Exit", 160, 140)],
  [Button("Tutorial", 160, 80), Button("Game", 160, 120), Button("Return", 160, 160)],
  [Button("Easy", 160, 80), Button("Medium", 160, 120), Button("Hard", 160, 160), Button("Return", 160, 200)],
  [Button("Return", 10, 10, Justify.TOPLEFT)],
  [Button("Return", 10, 10, Justify.TOPLEFT)]
]

# interval between icons, in ms
bpm = 190
note_interval = round((60/bpm) * 1000)
# in frames, assuming 60 fps
note_interval_frames = (note_interval/1000) * 60
# speed of icons, in pixels/frame
ICON_MOVE_SPEED = 5

# for timing...
icon_y_start = 180 - 3 * note_interval_frames * ICON_MOVE_SPEED

# ============================= GLOBAL VARIABLES =================
# which UI screen / game state we are in
# 0 = start menu, 1 = select gamemode, 2 = select tutorial difficulty, 3 = tutorial video / self-guided training, 4 = game
game_state = 0
# font for text displays
my_font = None
# pygame screen
screen = None
# whether the program is running (set to false to exit gracefully)
running = True
# score for game mode
score = 0

# key positions for game mode
keys_size = 40
keys_y = 180
keys_x = [20 + 60 * i for i in range(5)]
key_feedbacks = {key: ('', 0) for key in keys_x}
feedback_duration = 500

# (x,y,COLOR) of every moving icon above every key (y initially icon_y_start, x is one of the above key's x)
icons = []
# time at which last icon note was generated
last_note_time = pygame.time.get_ticks()

# pygame clock for timing
clock = pygame.time.Clock()

# data for tracking drum hits
shortest_hit = 0.05 # in seconds
LED_on_time = 0.05
last_hit_time = [0,0,0,0,0]
sounds = None
chan = None

# list of videos (easy, medium, hard)
videos = [None,None,None]

# drum hit events
drum_event_type = pygame.USEREVENT + 1
drum_hit_events = [pygame.event.Event(drum_event_type, which_drum=i) for i in range(5)]

# tempo events
tempo_event = pygame.USEREVENT + 2
pygame.time.set_timer(tempo_event, note_interval)

# ======================= FUNCTIONS ==============================

shutdown = False

def init_GPIO():
  # GPIO setup
  GPIO.setmode(GPIO.BCM)
  for pin in LED_PINS:
    GPIO.setup(pin, GPIO.OUT)
    GPIO.output(pin, GPIO.LOW)
  GPIO.setup(QUIT_BUTTON, GPIO.IN, pull_up_down=GPIO.PUD_UP)
  GPIO.setup(RETURN_BUTTON, GPIO.IN, pull_up_down=GPIO.PUD_UP)
  GPIO.setup(SHUTDOWN_BUTTON, GPIO.IN, pull_up_down=GPIO.PUD_UP)
  # add callback for quit button
  def GPIO_callback(channel):
    global running,video_playing
    print("Quitting")
    running = False
    video_playing=False
    os.system("echo 'quit' > /home/pi/final/ece5725project/mplayer_fifo")
    print("Stop playing")
  def GPIO_RETURN_callback(channel):
    os.system("echo 'quit' > /home/pi/final/ece5725project/mplayer_fifo")
    global game_state
    game_state=2
    print("Stop playing")
  def GPIO_SHUTDOWN_callback(channel):
    global shutdown, running
    running = False
    shutdown = True
    screen.fill(BLACK)
    pygame.display.flip()
    os.system("sudo shutdown -h now")
    print("shutdown")

  GPIO.add_event_detect(QUIT_BUTTON, GPIO.FALLING, callback=GPIO_callback, bouncetime=300)
  GPIO.add_event_detect(RETURN_BUTTON, GPIO.FALLING, callback=GPIO_RETURN_callback, bouncetime=300)
  GPIO.add_event_detect(SHUTDOWN_BUTTON, GPIO.FALLING, callback=GPIO_SHUTDOWN_callback, bouncetime=300)

def init_ADC():
  # create the spi bus
  spi = busio.SPI(clock=board.SCK_1, MISO=board.MISO_1, MOSI=board.MOSI_1)
  # create the cs (chip select)
  cs = digitalio.DigitalInOut(board.D5) # D22
  # create the mcp object
  mcp = MCP.MCP3008(spi, cs)
  # create 5 analog input channels (5 drums)
  global chan
  chan = [AnalogIn(mcp, p) for p in [MCP.P0, MCP.P1, MCP.P2, MCP.P3, MCP.P4]]

def setup_pygame():
  # init sound (https://stackoverflow.com/questions/18273722/pygame-sound-delay)
  pygame.mixer.pre_init(44100, -16, 2, 1024)
  pygame.mixer.init()
  if (ON_THE_PI):
    os.putenv('SDL_FBDEV', '/dev/fb0') # or fb1 if screen is connected
    os.putenv('SDL_VIDEODRIVER', 'fbcon')
    os.putenv('SDL_MOUSEDRV', 'TSLIB') # track mouse clicks on piTFT
    os.putenv('SDL_MOUSEDEV', '/dev/input/touchscreen')
    pygame.init()
    pygame.mouse.set_visible(False)
  else:
    pygame.init()
    pygame.mouse.set_visible(True)
  # setup screen
  global screen
  screen = pygame.display.set_mode((screen_width, screen_height))
  pygame.display.set_caption("Rhythm Game Menu")
  # initialize font
  pygame.font.init()
  global my_font
  # my_font = pygame.font.Font("SF-Pro.ttf", 24)
  # my_font = pygame.font.SysFont("segoeuisymbol", 24)
  my_font = pygame.font.SysFont(None, 24)

def load_sounds():
  # Load different sound for each drum
  global sounds
  sounds = [
    pygame.mixer.Sound('/home/pi/final/ece5725project/snare.wav'),
    pygame.mixer.Sound('/home/pi/final/ece5725project/cymballeft.wav'),
    pygame.mixer.Sound('/home/pi/final/ece5725project/tom1.wav'),
    pygame.mixer.Sound('/home/pi/final/ece5725project/cymbalright.wav'),
    pygame.mixer.Sound('/home/pi/final/ece5725project/tom2.wav')
  ]

def load_song(name='/home/pi/final/ece5725project/test1.mp3'):
  # load mp3 file
  try:
    pygame.mixer.music.load(name)
  except pygame.error:
    print("Error loading music file")
    sys.exit()

# draw text
def draw_text(surface, text, color, rect):
  text_surface = my_font.render(text, True, color)
  surface.blit(text_surface, rect)

# thread task to read ADC and play sounds when drum hit
def adc_task():
  while running:
    # detect drum hits
    for i in range(5):
      time_elapsed = time.time() - last_hit_time[i]
      if (chan[i].value > 3000 and time_elapsed > shortest_hit):
        # detect a drum hit
        pygame.event.post(drum_hit_events[i])
        last_hit_time[i] = time.time()
        for sound in sounds:
          sound.stop()
        sounds[i].play()
        GPIO.output(LED_PINS[i], GPIO.HIGH)
      # turn off LED if time elapsed
      elif time_elapsed > LED_on_time:
        GPIO.output(LED_PINS[i], GPIO.LOW)
    # slow down polling speed to 1 kHz
    sleep(0.001)

# ========================= INITIALIZATION =========================
init_GPIO()
init_ADC()
setup_pygame()
load_sounds()
load_song()
# create thread to read adc asynchronously
adc_thread = Thread(target=adc_task)
adc_thread.start()

# ========================= MAIN GAME LOOP =========================
while running:
  # ============================ DRAW ============================
  screen.fill(BLACK)

  if (shutdown):
    break

  # Draw buttons for the current menu level (for states 0, 1, 2)
  for button in buttons[game_state]:
    text = button.text
    pos = (button.x, button.y)
    text_surface = my_font.render(text, True, WHITE)
    if (button.justify == Justify.CENTER):
      rect = text_surface.get_rect(center=pos)
    elif (button.justify == Justify.TOPLEFT):
      rect = text_surface.get_rect(topleft=pos)
    screen.blit(text_surface, rect)

  # Draw other screen elements (for states 3, 4)
  if (game_state == 3):
    # TODO play video
    pass
  elif (game_state == 4): # display game UI
    # draw keys
    for x in keys_x:
      pygame.draw.rect(screen, WHITE, (x, keys_y, keys_size, keys_size))
    # draw moving icons
    for x, y, color in icons:
      pygame.draw.rect(screen, color, (x, y, keys_size, keys_size))
    # draw score text
    draw_text(screen, f"Score: {score}", WHITE, (10, 30))
    # draw title
    draw_text(screen, "Can't Even Get A POSITIVE Score? o_O", WHITE, (10, 50)) # 👈🤣
    draw_text(screen, "The record is 2200", WHITE, (100, 70))

  pygame.display.flip()

  # ============================ EVENTS ============================
  # event detection, state transition
  for event in pygame.event.get():
    if event.type == pygame.QUIT:
      running = False
    elif event.type == pygame.MOUSEBUTTONDOWN:
      pos = pygame.mouse.get_pos()
      for button in buttons[game_state]:
        text = button.text
        # detect collision
        if button.rect.collidepoint(pos):
          # state transition logic
          if (game_state == 0):
            if text == 'Exit':
              running = False
            elif text == 'Start':
              game_state = 1
          elif (game_state == 1):
            if text == 'Return':
              game_state = 0
            elif text == 'Tutorial':
              game_state = 2
            elif text == 'Game':
              game_state = 4
              pygame.mixer.music.play(loops=-1,start=0) # play the song for game state
          elif (game_state == 2):
            if text == 'Return':
              game_state = 1
            elif text in ['Easy', 'Medium', 'Hard']:
              game_state = 3
              # os.system("sudo SDL_VIDEODRIVER=fbcon SDL_FBDEV=/dev/fb0 mplayer -vf scale -zoom -xy 320 -vo sdl -framedrop tutorial_hard.mp4")
              selection = ['Easy', 'Medium', 'Hard'].index(text)
              video_filename=['tutorial_easy_320p.mp4', 'tutorial_medium_320p.mp4', 'tutorial_hard_320p.mp4']
              video_path = '/home/pi/final/ece5725project/' + video_filename[selection]
              subprocess.run(['mplayer', '-vo', 'fbdev2:/dev/fb0', '-input', 'file=/home/pi/final/ece5725project/mplayer_fifo', video_path])
          elif (game_state == 3):
            if text == 'Return':
              game_state = 2
          else: # game_state == 4
            if text == 'Return':
              game_state = 1
              pygame.mixer.music.rewind()
              pygame.mixer.music.stop()
            pass
    elif event.type == drum_event_type:
      which_drum = event.which_drum
      print(f"drum event detected, i={which_drum}")
      # light up drum LED
      # GPIO.output(LED_PINS[which_drum], GPIO.HIGH)
      # handle drum event for game
      if (game_state == 4):
        for i, (x, y, color) in enumerate(icons):
          # look for an icon in the right column (=x)
          if (x == keys_x[which_drum]):
            # check if in y range (hit/miss)
            if (screen_height - 100 <= y <= screen_height - 50):
              score += 10  # increase score
              icons.pop(i)  # Remove the slider for scoring
              feedback_message='Perfect!'
              key_feedbacks[keys_x[which_drum]] = ('Perfect!', pygame.time.get_ticks())
            else:
              score -= 5  # Decrease in score
              icons.pop(i)  # Remove unscored sliders
              feedback_message='Miss!'
              key_feedbacks[keys_x[which_drum]] = ('Miss!', pygame.time.get_ticks())
            break
        print(f"Score: {score}")
    elif event.type == tempo_event and game_state == 4:
      num_icons = random.randint(1, 3)
      columns = random.sample(list(range(5)), k=num_icons)
      for i in columns:
        x = keys_x[i]
        color = COLORS[i]
        icons.append((x, icon_y_start, color))

  # ============================ GAME STATE ============================
  
  # Update icon location (move down)
  if (game_state == 4):
    icons = [(x, y + ICON_MOVE_SPEED, color) for x, y, color in icons if y < screen_height]
  current_time = pygame.time.get_ticks()
  for i, (feedback, timestamp) in key_feedbacks.items():
    if feedback and (current_time - timestamp < feedback_duration):
      draw_text(screen, feedback, RED if feedback == 'Miss!' else GREEN, (i, keys_y))
    else:
      key_feedbacks[i] = ('', 0)  
  pygame.display.flip()
  clock.tick(60)

for pin in LED_PINS:
  GPIO.output(pin, GPIO.LOW)
pygame.quit()
adc_thread.join()